1e.
Is the UPC prepared to provide the same quality of
support to the newly engaged communities as it
provides to its current constituents?
While the support for all users will
remain at a very high level, that does not mean it will be exactly
the same. For example, for
the core community Unidata provides comprehensive
support for a full suite of tools from data services, through
decoders, to complete analysis and display
packages. For other cases, the tools that are
specialized to their community may not be
available via and supported by the UPC. One example of this is
the community of users of GIS tools. In that case
Unidata supports standards-based web services that make our datasets
available in such a way that tools that incorporate those standard
interfaces can avail themselves of Unidata datasets.
Thus these new communities can continue to make use of the
analysis and display tools they are familiar with while taking
advantage of the data services of the traditional
Unidata community.
Excerpt from the proposal review panel report
Advocacy for Community Standards:
"In particular, the UPC could play a significant leadership
role within committees and consortiums like OGC seeking to
address the need to develop standards and technologies for data
discovery. Unidata leadership and advocacy in this area could
facilitate expanded utilization of Unidata information
resources for other research areas like climate and provide
Unidata users with easier access to other data sources like
NASA satellite information. However, the OGC letter of
recommendation in the proposal and the Unidata responses to the
review panel questions regarding cyberinfrastructure did
demonstrate that the Unidata was actively involved in community
discussion of interface and data standards."
Relationship
to Current Unidata Strategic Plan
Below are a few excerpts from the current Unidata Strategic
Plan that highlight the importance of the outreach activities
summarized in this status update?
-
... to build infrastructure that makes it easy to integrate
and use data from disparate geoscience disciplines
-
Data formats like netCDF, together with community-based
data standards like the Climate and Forecast metadata
convention and the Common Data Model are enhancing the
widespread usability and interoperability of scientific
datasets.
-
Advance geoscience data and metadata standards and
conventions
-
... close partnerships and collaboration with geoscience
data providers, tool developers, and other stakeholders,
-
... our experience shows us that robust solutions arise
from community and collaborative efforts
-
... close partnerships and collaboration with geoscience
data providers, tool developers, and other stakeholders, and
the informed guidance of our governing committees will all
be important catalysts for Unidata’s success.
Summary
of Recent Progress
Progress on OGC standardization
As
the official UCAR representative to the OGC Technical
Committee, Unidata participates in 3-4 technical committee
meetings per year to ensure that Unidata and UCAR needs
are met in the emerging international standards.
In 2011, the netCDF Classic data
model was established as the OGC core netCDF standard. The
binary encoding for the classic data model was established as the
first extension to the netCDF core standard. Since the last
Policy Committee report, the netCDF enhanced data model and the CF
(Climate and Forecast) conventions have been formally adopted as
extensions
to the netCDF core standard. The OGC-adopted
standards documents are available at
- http://www.opengeospatial.org/standards/netcdf
This completes the primary objectives we had laid out for the
CF-netCDF standards initiative in the OGC. However, the
CF-netCDF Standards Working Group (SWG) is also considering ncML
(netCDF Markup Language) as an XML encoding format for netCDF.
In addition, a new initiative for encoding uncertainty information has
been formally adopted as an OGC Discussion Paper.
- http://www.opengeospatial.org/node/1778
New Issue
At
the most recent OGC Technical Committee meetings, the Coverages
DWG (Domain Working Group), the WCS (Web Coverage Service) SWG
(Standards Working Group), and the CF-netCDF SWG had several
"lively" and hopefull productive discussions regarding how to
incorporate coverage encodings (e.g., geoTIFF, JPEG2000, netCDF)
into OGC protocol specs (not just WCS but also possibly WFS, SOS,
WPS, ...). There appears to be general agreement that, as
much as possible, these coverage encoding specifications (e.g.,
the encoding data model mappings to GMLCOV and the special
parameters for each binary encoding) should be decoupled from the
data access protocols. If you recall, we had been proposing
the CF-netCDF specifically as an encoding for WCS 2.0.
So
far several ideas have been proposed for how to go about
doing this but, as yet, there is no clear agreed-upon path.
During these discussions, it was noted that the OGC
Architecture Board is also considering ways to streamline and
simplify some of the rather rigid requirements for how the
specification documents are written. Some possible
mechanisms are aiming for breaking specs into fewer modules,
providing a cleaner and less distracting means for dealing with
HTTP URI requirements of the OGC Naming Authority, and perhaps
less emphasis and dependence on UML diagrams. Having spent
a large fraction of my time writing and rewriting those portions
of the existing CF-netCDF documents, I think these are moves in
the right direction.
Ongoing
Outreach Activities
Earthcube Activities.
For
Unidata Outreach at least, EarthCube activities have become
less dominant
Interactions continue in two areas that started as EarthCube Concept
Awards: Brokering and Cross-domain Interoperability. Our
work in these projects is valuable inside or outside EarthCube because
it is concentrated on making our data more discoverable and
accessible. One key element is to work with a third,
middle tier between clients and servers in the web services
architecture. This layer will make metadata from THREDDS
Data Servers more readily available to a variety of data discovery
systems and will make the datasets themselves more conveniently
accessible via many protocols not supported in the TDS
itself. We are working with groups developing software for
this brokering tier. One such product is the open source
ESRI GeoPortal which has been the primary target recent "hackathons"
in which several of TDS servers were involved. Another is a
joint effort with the University of Florence ESSI Lab to experiment
with their web services brokering layer tools to determine whether we
can simplify our web client and server tools by using a brokering
layer to do transformations among metadata and data service protocols
and encodings. Visitors from ESSI Labs, including the director,
spent much of a week visiting Unidata and interacting with staff on
these topics as well as OGC standardization and possible
collaborations with the Google Earth Engine team.
OGC Standards Actions
- Enhanced (netCDF4) data model adopted
as OGC extension standard to netCDF core.
- CF conventions adopted
as OGC extension standard to netCDF core.
- CF-netCDF encoding for Web Coverage Service has been drafted
- OGC discussion initiated on best mechanism for connecting
CF-netCDF encoding to various OGC service protocols
- Dicusson Paper published on Uncertainty Conventions for netCDF
- OPenDAP access protocol needs to be coordinated
- HDF encoding needs to be coordinated
New and Ongoing Collaborations:
- NCAR GIS Program
- Liaison with international ODIP (Ocean Data Interoperability
Platform) project
- Collaboration with ESSI Labs to experiment with their
brokering layer in conjunction with THREDDS Data Servers
- Marine Metadata Interoperability (MMI) Project Steering Team
- CUAHSI Standing Committee
- UCAR wide representative to OGC Technical Committee
- AGU ESSI Focus Group Board
- ESIN Journal Editorial Board
- Liaison to OOI Cyberinfrastructure Project
- Several collaborations with EarthCube teams
- Collaborative European / US effort on the Ocean Data
Interoperability Platform
- Potential collaboration with SDSC team on annotating datasets
with information gained from support archives
- Experimentation with ESSI Labs on use of brokering tier web
services technology
Since the last Policy Committee meeting, the ODIP (Ocean
Data Interoperability Platform) was funded by the European
Commission and we continue to work with San Diego Supercomputing
Center and Woods Hole to get the US part of the project funded by
NSF. Unidata's technologies (especially THREDDS and netCDF)
are part of the project and we also maintain a liaison role to
make out community aware of the work an possible
applications. Unidata was invited to participate in the
initial workshop in February.
- http://www.odip.org/content/news_details.asp?menu=0100000_000001
-
- http://seadatanet.maris2.nl/newsletter.asp#70
-
Planned
Activities
The next step in the
CF-netCDF standardization is to work with others to come up with
a general solution to the issue of how coverage encoding
specifications should be related to the various OGC web service
specifications. However, it should be noted that, with
CF-netCDF established as an international OGC encoding standard,
the primary objectives have been accomplished. The
discussion paper on netCDF conventions for encapsulating
uncertainty information has been approved and is under active
discussion whose outcome will determine whether this will
eventually be proposed as an additional extension to the netCDF
core standard. Work is likely to accelerate on collaborations
with OPeNDAP and HDF who are now active in the OGC.
An approach for dealing with the
HDF5 encoding of the netCDF enhanced data model is still being
sought.
The effort to establish CF-netCDF as an encoding format for WCS
(as well as WFS, WPS, and SOS in the long term) has led to an
OGC discussion about a more general mechanism for establishing
encoding formats for multiple OGC data access
protocols. Unidata is participating actively in
these developments.
Based on an earlier policy committee meeting presentation, I
created a white paper based on my "Data Interactive
Publications" presentation which seemed to be well
received. It's available at
- https://sites.google.com/site/datainteractivepublications/home/white-paper-on-data-interactive-publications
Considerable support for this concept developed at the charrette
and the concept was moved forward by a team lead by Tanu Malik of
the University of Chicago. However, it was not among
the Expressions of Interest encouraged to submit an EAGER
proposal. The group is considering publishing an
article based on the EarthCube whitepaper and subsequent work in
an online journal.
A follow up presentation on was presented as a keynote at the
triennial Unidata User Workshop. Discussion of this topic
surfaces now and then. Most recently it came up in our
interactions with Google about possible future collaborations.
Relevant
Metrics
- Two
more netCDF-related OGC international standards
(netCDF 4 data model and CF conventions)
-
The list of "other collaborations" above includes a
dozen organizations we have regular interactions
with. In most cases, our interactions are as
representatives of our community on their steering or
policy groups, so we have at least some voice in their
direction.
- One
additional international collaboration (ODIP)
- One
new potential collaboration with industry (Google
Earth Engine).
-
Over the recent years of these standardization
efforts, ESRI has incorporated the netCDF among the
input and output formats that their arcGIS tools work
with directly. This represents a user community
that numbers in the millions, but it isn't possible
for us to measure how many of those users now use it
to access our data.
-
The standards efforts enable us to collaborate on an
ongoing basis with dozens of international
organizations -- especially those represented in the
OGC MetOceans, Earth System Science, and Hydrology
Domain Working Groups.
